Databases may be used to organize collections of data. Further, analytics systems (e.g., cloud based data analytics systems) may be used to analyze the data (e.g., operational statistics, metadata, parameters, etc.) captured in the database. In certain cases, a single database may store data from a number of different data sources (e.g., homogenous or heterogeneous data sources, such as applications, servers, live data streams, virtual computing instances, etc.). For example, it may be desirable to correlate data from different data sources when performing analytics on the data, which may be easier to perform if the data from the multiple data sources is stored in the same database.
The data from the various data sources may be extracted from the data sources, transformed, and loaded into the database on a continuous basis using continuous extract, transform, load (ETL) processes as the various data sources may continually produce new data, remove data, update data, etc. For example, in an extract operation, data is extracted from the data source. Further, in a transform operation, a series of rules or functions are applied to the extracted data so it can be loaded into the target database. For example, the extracted data may be transformed by selecting only certain columns of data to load in the database, encoding the data into other values, translating values, calculating values based on the data, etc. so that the transformed data matches with a database schema (e.g., structure of the database, such as tables (including rows and columns), fields, relationships, functions, etc.) of the database and can be properly loaded in the database. In the load operation, the transformed data is loaded into the database itself.
Further, the database schema of the database may be dynamic and change during operation of the database. For example, new tables and columns may be created/removed in the database as data from new data sources is added to or old data sources removed from the database. Since the database schema of the database may change during operation of the database data consistency may need to be ensured, such that data entered into the database from ETL processes matches the current database schema and not an outdated database schema.